{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "92cf0c6c",
      "metadata": {
        "id": "92cf0c6c"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/pinecone-import.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/docs/pinecone-import.ipynb)\n",
        "\n",
        "# Import from object storage\n",
        "\n",
        "**Note:** This feature is in [public preview](https://docs.pinecone.io/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2LIjHbuAa4fQ",
      "metadata": {
        "id": "2LIjHbuAa4fQ"
      },
      "source": [
        "## Scenario: Ingesting Parquet Data from S3 to Pinecone Serverless\n",
        "\n",
        "In this scenario, we will be generating JSON data, embedding that data using the [Pinecone Inference API](https://docs.pinecone.io/guides/inference/understanding-inference), storing the data in AWS S3 as Parquet files, and ingesting the data from S3 into a Pinecone Serverless index.\n",
        "\n",
        "### Problem Overview\n",
        "\n",
        "The goal is to move the data from S3 to Pinecone so that it can be used for future tasks such as semantic search. This process ensures that the data is efficiently searchable and retrievable by applications.\n",
        "\n",
        "### Solution Steps\n",
        "\n",
        "1. **Generate data**: Begin by generating the data that needs to be processed.\n",
        "\n",
        "2. **Chunk data**: Split the generated data into smaller, manageable chunks that can be processed and embedded effectively.\n",
        "\n",
        "3. **Embed data**: Create vector embeddings from the chunked data. These embeddings are crucial for indexing and retrieval in Pinecone.\n",
        "\n",
        "4. **Create Parquet files**: Save the vector embeddings, along with metadata, into Parquet files.\n",
        "\n",
        "5. **Access S3 bucket**: Access the S3 bucket where the Parquet files will be stored.\n",
        "\n",
        "6. **Upload Parquet files**: Upload the Parquet files containing the embeddings to the S3 bucket.\n",
        "\n",
        "7. **Create Pinecone index**: Create a Pinecone index where the embeddings will be stored. This index will allow for efficient similarity search and other tasks.\n",
        "\n",
        "8. **Load S3 data into Pinecone index**: Load the embeddings from the S3 bucket into the Pinecone index.\n",
        "\n",
        "\n",
        "Please see our official [Understanding Imports in Pinecone documentation](https://docs.pinecone.io/guides/data/understanding-imports) for additional information.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "kdJjv4CHJOwA",
      "metadata": {
        "id": "kdJjv4CHJOwA"
      },
      "source": [
        "The data flow for the notebook is outlined below:"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DP3Q0AhuXRQ9",
      "metadata": {
        "id": "DP3Q0AhuXRQ9"
      },
      "source": [
        "![Pinecone101_flow.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5136421b",
      "metadata": {
        "id": "5136421b"
      },
      "source": [
        "## Import required libraries"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "698bf926",
      "metadata": {
        "id": "698bf926"
      },
      "source": [
        "First, we need to import all the necessary libraries that will be used for phrase generation, text chunking, embedding, and uploading to S3."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "58502fd1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "58502fd1",
        "outputId": "0df49eba-c481-43a1-cfd5-b11e64a8ee25"
      },
      "outputs": [],
      "source": [
        "!pip install pinecone\n",
        "!pip install pinecone_notebooks\n",
        "!pip install langchain\n",
        "!pip install boto3\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7407f360",
      "metadata": {
        "id": "7407f360"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import pandas as pd\n",
        "import boto3\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "from pinecone import Pinecone, ServerlessSpec\n",
        "import os"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c641c107",
      "metadata": {
        "id": "c641c107"
      },
      "source": [
        "## Generate unique phrases"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1a0f82e4",
      "metadata": {
        "id": "1a0f82e4"
      },
      "source": [
        "In this step, we create a list of adjectives, nouns, and verbs, and then randomly combine them to form 100 unique phrases. Each phrase will follow the structure: 'The [adjective] [noun] [verb] over the [adjective] [noun]'. These phrases will be stored in a Pandas DataFrame for further processing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "91591bb0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "id": "91591bb0",
        "outputId": "97c903d9-38e3-4279-e1c8-9dc108eae6b3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Generating 100 unique phrases...\n",
            "Unique phrases generated.\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 100,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 100,\n        \"samples\": [\n          \"id83\",\n          \"id53\",\n          \"id70\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 100,\n        \"samples\": [\n          \"The quick lion runs over the dark bird\",\n          \"The bright lion flies over the dark sky\",\n          \"The lazy bird sits over the wild fox\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "df"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-bc5153ee-b71c-49eb-9298-3352f2f3bcd1\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>id0</td>\n",
              "      <td>The brave tiger roars over the silent river</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>id1</td>\n",
              "      <td>The bold mountain observes over the mysterious...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>id2</td>\n",
              "      <td>The bright tree roars over the quick bird</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>id3</td>\n",
              "      <td>The bold lion observes over the lazy river</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>id4</td>\n",
              "      <td>The mysterious bird climbs over the quick tree</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bc5153ee-b71c-49eb-9298-3352f2f3bcd1')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-bc5153ee-b71c-49eb-9298-3352f2f3bcd1 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-bc5153ee-b71c-49eb-9298-3352f2f3bcd1');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-7a4992d0-eae1-46e3-8bc3-e57636a42d99\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7a4992d0-eae1-46e3-8bc3-e57636a42d99')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-7a4992d0-eae1-46e3-8bc3-e57636a42d99 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "    id                                             values\n",
              "0  id0        The brave tiger roars over the silent river\n",
              "1  id1  The bold mountain observes over the mysterious...\n",
              "2  id2          The bright tree roars over the quick bird\n",
              "3  id3         The bold lion observes over the lazy river\n",
              "4  id4     The mysterious bird climbs over the quick tree"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Lists of words to combine into unique phrases\n",
        "adjectives = ['quick', 'lazy', 'energetic', 'bright', 'dark', 'mysterious', 'bold', 'silent', 'wild', 'brave']\n",
        "nouns = ['fox', 'dog', 'cat', 'tree', 'river', 'mountain', 'bird', 'sky', 'lion', 'tiger']\n",
        "verbs = ['jumps', 'runs', 'flies', 'sits', 'sleeps', 'roars', 'whispers', 'climbs', 'chases', 'observes']\n",
        "\n",
        "# Generate 100 unique phrases\n",
        "num_rows = 100\n",
        "print(f\"Generating {num_rows} unique phrases...\")\n",
        "\n",
        "# Create a DataFrame with unique phrases\n",
        "data = [{'id': f'id{i}', 'values': f\"The {random.choice(adjectives)} {random.choice(nouns)} {random.choice(verbs)} over the {random.choice(adjectives)} {random.choice(nouns)}\"} for i in range(num_rows)]\n",
        "df = pd.DataFrame(data)\n",
        "print(\"Unique phrases generated.\")\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1c25b6ca",
      "metadata": {
        "id": "1c25b6ca"
      },
      "source": [
        "## Chunk text using LangChain"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9ebfb42f",
      "metadata": {
        "id": "9ebfb42f"
      },
      "source": [
        "Here we chunk the generated phrases into smaller pieces using LangChain's `RecursiveCharacterTextSplitter`. This is useful when working with large texts, as smaller chunks can be processed more efficiently."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8edf02bd",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "8edf02bd",
        "outputId": "964121dd-deb2-4ba1-d379-505f3d1b5979"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"df_chunked\",\n  \"rows\": 100,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 100,\n        \"samples\": [\n          \"id83\",\n          \"id53\",\n          \"id70\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 100,\n        \"samples\": [\n          \"The quick lion runs over the dark bird\",\n          \"The bright lion flies over the dark sky\",\n          \"The lazy bird sits over the wild fox\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "df_chunked"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-2bb58391-8ec0-4ad5-80e3-7a0cf54adaf4\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>id0</td>\n",
              "      <td>The brave tiger roars over the silent river</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>id1</td>\n",
              "      <td>The bold mountain observes over the mysterious...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>id2</td>\n",
              "      <td>The bright tree roars over the quick bird</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>id3</td>\n",
              "      <td>The bold lion observes over the lazy river</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>id4</td>\n",
              "      <td>The mysterious bird climbs over the quick tree</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2bb58391-8ec0-4ad5-80e3-7a0cf54adaf4')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-2bb58391-8ec0-4ad5-80e3-7a0cf54adaf4 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-2bb58391-8ec0-4ad5-80e3-7a0cf54adaf4');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-61bae143-441d-48ff-a412-f4b15964fb3e\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-61bae143-441d-48ff-a412-f4b15964fb3e')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-61bae143-441d-48ff-a412-f4b15964fb3e button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "    id                                             values\n",
              "0  id0        The brave tiger roars over the silent river\n",
              "1  id1  The bold mountain observes over the mysterious...\n",
              "2  id2          The bright tree roars over the quick bird\n",
              "3  id3         The bold lion observes over the lazy river\n",
              "4  id4     The mysterious bird climbs over the quick tree"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Chunk the text column using LangChain's RecursiveCharacterTextSplitter\n",
        "chunk_size = 1000\n",
        "chunk_overlap = 100\n",
        "\n",
        "# Initialize the text splitter\n",
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)\n",
        "\n",
        "# Perform the chunking process and store the chunked texts\n",
        "chunked_texts = []\n",
        "\n",
        "for text in df['values']:\n",
        "    if pd.isna(text) or text == '':\n",
        "        chunked_texts.append([text])  # Keep empty texts as they are\n",
        "    else:\n",
        "        chunked_text = text_splitter.split_text(text)\n",
        "        chunked_texts.append(chunked_text)\n",
        "\n",
        "# Flatten the DataFrame by exploding the chunked texts into separate rows\n",
        "df['values'] = chunked_texts\n",
        "df_chunked = df.explode('values').reset_index(drop=True)\n",
        "df_chunked.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "n6Y-EX3BgB0z",
      "metadata": {
        "id": "n6Y-EX3BgB0z"
      },
      "source": [
        "## Get your API key"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cb6be6ec",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 247
        },
        "id": "cb6be6ec",
        "outputId": "12ffc9cf-47be-45ce-e31d-0670c5af09c4"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<script type=\"text/javascript\" src=\"https://connect.pinecone.io/embed.js\"></script>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from pinecone_notebooks.colab import Authenticate\n",
        "Authenticate()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "63db8017",
      "metadata": {
        "id": "63db8017"
      },
      "source": [
        "## Initialize a Pinecone client"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "831325b5",
      "metadata": {
        "id": "831325b5"
      },
      "source": [
        "We load the Pinecone API key from a configuration file (`config.ini`) and initialize the Pinecone client. Pinecone will be used to embed the chunked text."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "LKL1_B6GfA69",
      "metadata": {
        "id": "LKL1_B6GfA69"
      },
      "outputs": [],
      "source": [
        "api_key = os.getenv('PINECONE_API_KEY')\n",
        "\n",
        "# Configure Pinecone client\n",
        "pc = Pinecone(api_key=api_key)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5dc26203",
      "metadata": {
        "id": "5dc26203"
      },
      "source": [
        "## Embed text using the Pinecone Inference API"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "21b7050c",
      "metadata": {
        "id": "21b7050c"
      },
      "source": [
        "Next, we embed the chunked text using Pinecone's embedding model. We process each chunk and store the embedding values back into the DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dd938b95",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 224
        },
        "id": "dd938b95",
        "outputId": "94c35fc7-9e9d-4f4b-c036-da7be34d12ee"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "All embeddings generated.\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"df_chunked\",\n  \"rows\": 100,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 100,\n        \"samples\": [\n          \"id83\",\n          \"id53\",\n          \"id70\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"values\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "df_chunked"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-16b7d167-80b5-494e-88b8-52ba29ef76f2\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>values</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>id0</td>\n",
              "      <td>[-0.00984954833984375, 0.0060577392578125, -0....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>id1</td>\n",
              "      <td>[0.0022792816162109375, 0.0202178955078125, -0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>id2</td>\n",
              "      <td>[0.0230560302734375, 0.02764892578125, -0.0150...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>id3</td>\n",
              "      <td>[-0.01027679443359375, 0.0150299072265625, -0....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>id4</td>\n",
              "      <td>[0.0034313201904296875, 0.016632080078125, -0....</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-16b7d167-80b5-494e-88b8-52ba29ef76f2')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-16b7d167-80b5-494e-88b8-52ba29ef76f2 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-16b7d167-80b5-494e-88b8-52ba29ef76f2');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-df403127-adae-4fcb-a106-a5f35bc29a8e\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-df403127-adae-4fcb-a106-a5f35bc29a8e')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-df403127-adae-4fcb-a106-a5f35bc29a8e button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "    id                                             values\n",
              "0  id0  [-0.00984954833984375, 0.0060577392578125, -0....\n",
              "1  id1  [0.0022792816162109375, 0.0202178955078125, -0...\n",
              "2  id2  [0.0230560302734375, 0.02764892578125, -0.0150...\n",
              "3  id3  [-0.01027679443359375, 0.0150299072265625, -0....\n",
              "4  id4  [0.0034313201904296875, 0.016632080078125, -0...."
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Function to get embeddings for a given text\n",
        "def get_embedding(text):\n",
        "    res = pc.inference.embed(\n",
        "        model=\"multilingual-e5-large\",\n",
        "        inputs=text,\n",
        "        parameters={\n",
        "            \"input_type\": \"query\",  # or \"passage\"\n",
        "            \"truncate\": \"END\"\n",
        "        }\n",
        "    )\n",
        "    embedding = res.data[0]['values']\n",
        "    return embedding\n",
        "\n",
        "# Embed the chunked phrases in the DataFrame\n",
        "for index, row in df_chunked.iterrows():\n",
        "    #print(f\"Processing record {index + 1}/{len(df_chunked)}: {row['id']}\")\n",
        "    df_chunked.at[index, 'values'] = get_embedding(row['values'])\n",
        "\n",
        "print(\"All embeddings generated.\")\n",
        "df_chunked.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4f627ebe",
      "metadata": {
        "id": "4f627ebe"
      },
      "source": [
        "## Upload embedded data to S3"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0110506c",
      "metadata": {
        "id": "0110506c"
      },
      "source": [
        "Now, we can upload the embedded DataFrame to an Amazon S3 bucket as Parquet files in chunks. This step uses the `boto3` library to interact with S3.\n",
        "\n",
        "Be sure to replace the bucket name and folder name."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "70dc8a47",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "id": "70dc8a47",
        "outputId": "32a50183-1ff4-4e45-ce65-dcf16381ee44"
      },
      "outputs": [],
      "source": [
        "# Function to convert the DataFrame to Parquet in-memory and upload to S3 in chunks\n",
        "def upload_to_s3(df, bucket, folder, chunk_size=10):\n",
        "    s3_client = boto3.client('s3', config=Config(signature_version=UNSIGNED))\n",
        "    total_rows = len(df)\n",
        "    total_chunks = (total_rows + chunk_size - 1) // chunk_size  # Calculate the total number of chunks\n",
        "\n",
        "    for i, start in enumerate(range(0, total_rows, chunk_size)):\n",
        "        chunk = df.iloc[start:start + chunk_size]\n",
        "\n",
        "        # Save the chunk as a Parquet file in memory\n",
        "        parquet_buffer = BytesIO()\n",
        "        chunk.to_parquet(parquet_buffer, index=False)\n",
        "        parquet_buffer.seek(0)\n",
        "\n",
        "        object_name = f\"{folder}/part-{i}.parquet\"\n",
        "        try:\n",
        "            # Upload the Parquet file from memory to S3\n",
        "            s3_client.put_object(Body=parquet_buffer, Bucket=bucket, Key=object_name)\n",
        "            print(f\"Part-{i}.parquet successfully uploaded to {bucket}/{object_name}\")\n",
        "        except ClientError as e:\n",
        "            print(f\"Failed to upload part-{i}.parquet: {e}\")\n",
        "\n",
        "bucket_name = \"rjbucketaws\"  # Replace with your bucket name\n",
        "folder_name = \"1-1024-1/namespace1\"  # Specify the folder name\n",
        "\n",
        "# Start the upload process\n",
        "upload_to_s3(df_chunked, bucket_name, folder_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c5e2cdc0",
      "metadata": {
        "id": "c5e2cdc0"
      },
      "source": [
        "## Create a serverless index\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8dcb7acb",
      "metadata": {
        "collapsed": true,
        "id": "8dcb7acb"
      },
      "outputs": [],
      "source": [
        "# Get cloud and region settings\n",
        "cloud = os.getenv('PINECONE_CLOUD', 'aws')\n",
        "region = os.getenv('PINECONE_REGION', 'us-east-1')\n",
        "\n",
        "# Define serverless specifications\n",
        "spec = ServerlessSpec(cloud=cloud, region=region)\n",
        "\n",
        "index_name = \"pinecone-import1-1024-1\"\n",
        "dimension = 1024\n",
        "\n",
        "# List all existing indexes\n",
        "existing_indexes = pc.list_indexes()\n",
        "\n",
        "# Check if the index exists\n",
        "if index_name not in existing_indexes.names():\n",
        "    # Create the index if it doesn't exist\n",
        "    pc.create_index(\n",
        "        name=index_name,\n",
        "        dimension=dimension,\n",
        "        metric=\"cosine\",\n",
        "        spec=ServerlessSpec(cloud=\"aws\", region=\"us-west-2\")\n",
        "    )\n",
        "    print(f\"Index '{index_name}' created successfully.\")\n",
        "else:\n",
        "    print(f\"Index '{index_name}' already exists.\")\n",
        "\n",
        "# Connect to the index\n",
        "index = pc.Index(name=index_name)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bfc374ca",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bfc374ca",
        "outputId": "51441002-6cd2-4711-82bc-400e2246a059"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'dimension': 1024,\n",
              " 'index_fullness': 0.0,\n",
              " 'namespaces': {},\n",
              " 'total_vector_count': 0}"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9719a558",
      "metadata": {
        "id": "9719a558"
      },
      "source": [
        "## Start import task\n",
        "\n",
        "Each file contains:\n",
        "\n",
        "*   **id**: Unique identifier\n",
        "*   **Values**: Embedded vectors\n",
        "*   **metadata**: JSON-formatted dictionary with metadata\n",
        "\n",
        "***Note***: *This task may take 10 minutes or more to complete. Each import request can import up 1TB of data, or 100,000,000 records into a maximum of 100 namespaces, whichever limit is met first.*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "73197a39",
      "metadata": {
        "id": "73197a39"
      },
      "source": [
        "## Specify AWS S3 folder and start task"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b979e842",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426
        },
        "id": "b979e842",
        "outputId": "a40bbc72-3eab-462e-aafe-39f5b5a59e28"
      },
      "outputs": [],
      "source": [
        "# Specify S3 data URI\n",
        "root = \"[YOUR S3 URI HERE]\"\n",
        "\n",
        "# Start import\n",
        "op = index.start_import(uri=root, error_mode=\"ABORT\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7a908310",
      "metadata": {
        "id": "7a908310"
      },
      "source": [
        "## Check the status of the import"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "936032f7",
      "metadata": {
        "id": "936032f7"
      },
      "outputs": [],
      "source": [
        "index.describe_import(id=op.id)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a56aced9",
      "metadata": {
        "id": "a56aced9"
      },
      "source": [
        "## List import operations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "eeac0925",
      "metadata": {
        "id": "eeac0925"
      },
      "outputs": [],
      "source": [
        "imports = list(index.list_imports())\n",
        "if imports:\n",
        "    for i in imports:\n",
        "        print(i)\n",
        "else:\n",
        "    print(\"No imports found.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8c0f1908",
      "metadata": {
        "id": "8c0f1908"
      },
      "source": [
        "## Describe a specific import"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a2dc3916",
      "metadata": {
        "id": "a2dc3916"
      },
      "outputs": [],
      "source": [
        "index.describe_import(\"1\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e9899dbd",
      "metadata": {
        "id": "e9899dbd"
      },
      "source": [
        "## Cancel the Import (if needed)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "49121d9b",
      "metadata": {
        "id": "49121d9b"
      },
      "outputs": [],
      "source": [
        "# Check if operation status and cancel running instance\n",
        "op_status = index.describe_import(op.id)\n",
        "print(f\"Operation status: {op_status}\")\n",
        "\n",
        "if op_status in ['in_progress', 'pending']:\n",
        "    try:\n",
        "        cancel_response = index.cancel_import(op.id)\n",
        "        print(f\"Import operation {op.id} cancelled.\")\n",
        "    except Exception as e:\n",
        "        print(f\"Error cancelling import: {e}\")\n",
        "else:\n",
        "    print(f\"Cannot cancel operation {op.id} because its status is: {op_status}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "28a43542",
      "metadata": {
        "id": "28a43542"
      },
      "source": [
        "## Delete the index"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0afe1809",
      "metadata": {
        "id": "0afe1809"
      },
      "outputs": [],
      "source": [
        "pc.delete_index(index_name)\n",
        "print(f\"Index '{index_name}' deleted.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a139b974",
      "metadata": {
        "id": "a139b974"
      },
      "source": [
        "## Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a9e55ea6",
      "metadata": {
        "id": "a9e55ea6"
      },
      "source": [
        "In this notebook, we successfully generated random phrases, chunked them, embedded the chunked texts using Pinecone, and uploaded the final embedded data to Amazon S3. You can further customize this notebook for you use case by updating it to use your S3 bucket, changing the chunk size or embedding model as needed."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
